feat(databases): add --url flag to tables load for remote parquet files#88
Conversation
Allows loading a remote parquet file directly by URL without a local download step. The CLI fetches the URL via reqwest, streams it to POST /files, and shows a progress bar (bytes+eta if Content-Length is present, spinner otherwise). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| use crossterm::style::Stylize; | ||
| eprintln!("{}", crate::util::api_error(resp_body).red()); | ||
| fn upload_parquet_url(api: &ApiClient, url: &str) -> String { | ||
| if !is_parquet_path(url) { |
There was a problem hiding this comment.
nit: is_parquet_path checks ends_with(".parquet") against the raw URL, so it rejects URLs with query strings or fragments — e.g. S3/GCS presigned URLs like https://bucket.s3.amazonaws.com/file.parquet?X-Amz-Signature=... would be turned away even though they point at a valid parquet file. Consider parsing the URL and validating only the path component (or stripping ?/# before the extension check) so signed URLs work. (not blocking)
|
|
||
| let body: serde_json::Value = match serde_json::from_str(&resp_body) { | ||
| Ok(v) => v, | ||
| let resp = match reqwest::blocking::get(url) { |
There was a problem hiding this comment.
super nit: reqwest::blocking::get uses a default client with no timeout, so a remote that accepts the TCP connection but never responds will hang the CLI indefinitely (only Ctrl-C will get out). Consider building a reqwest::blocking::Client with connect_timeout / timeout and using it here. (not blocking)
1 participant
Summary
--url <url>flag tohotdata databases tables load, mutually exclusive with--fileand--upload-id.parquetbefore fetchingPOST /filesusingreqwestblocking (already a project dependency) — no temp file on diskContent-Length; spinner fallback when notfinish_uploadhelper to deduplicate upload/error-handling logic between--fileand--urlpathsTest plan
hotdata databases tables load taxi yellow_tripdata --url https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2026-01.parquet--url+--filetogether are rejected by clap--url+--upload-idtogether are rejected by clapcargo test)🤖 Generated with Claude Code